Distributed file systems such as used in network storage systems suffer performance problems resulting from poor allocation of resources in response to changing workloads. The resources in this context include memory, processor cycles, and network bandwidth. In a healthy distributed file system, applications make input/output (I/O) requests to a storage server and receive a response to each request within acceptable latency limits. However, distributed file systems can be subject to enormous variations in demand. Performance problems typically arise as more capacity, more users, or more workload are added to existing systems. Additionally, file system clients generally operate selfishly, trying to maximize their own throughput, which stresses system resources. When a system resource becomes congested, the increase in latency can be sudden and exponential.
File system operations have different priorities implicitly. While some file system operations need to be performed on demand, many can be scheduled; these two classes of operations can be termed synchronous and asynchronous operations, respectively. Asynchronous operations include writes and read-aheads. With writes, the calling application is released once the write completes to local memory. Read-aheads are submitted by the file system and have no application awaiting their completion. In some distributed file system, asynchronous operations make up a substantial fraction (e.g., about one third) of all requests.
Current distributed file systems suffer from a priority inversion between synchronous and asynchronous operations, where asynchronous operations are actually given preference to synchronous operations. When this priority inversion occurs, an application that has issued a synchronous operation has to wait until previously issued asynchronous operations complete. This priority inversion adversely affects application—(client—) perceived performance, which is a problem that existing approaches fail to address.
Current resource management solutions tend to be limited to only isolated portions of the entire system. For example, quality of service (QoS) techniques encode priorities between clients or I/O streams in an effort to guarantee individual service level agreements (SLAs). They employ feedback-based I/O throttling in order to limit resource congestion and avoid throughput-based crashes. However, these approaches are based on server-oriented performance metrics only, such as I/Os per second (IOPS), I/O throughput, and network utilization, which in many cases do not correspond well with application-perceived (user-perceived) performance.
Server-oriented metrics are insensitive to application-perceived performance, because they fail to distinguish the urgency and relative priority of file system operations by I/O type or based on client state. From the server's perspective, all client operations at any given time are equally important, which is generally not true.
Congestion pricing is a technique sometimes used for flow control in network environments. However, distributed file systems present a richer, more complex environment, in that there are multiple different resources to manage. In distributed file systems, resources are heterogeneous: in most cases, no two resources are directly comparable. For example, one cannot directly balance CPU cycles against memory utilization or vice versa. Nor does either of these resources map naturally to network bandwidth. This makes the assessment of the load on a distributed system difficult. The performance-limiting factors in a distributed file system often depend on a complex combination of system configuration, workload, and the client population. As such, optimizing I/O systems or networks in isolation does not suffice.